add mnnvl pipeline using gb200 nodes by Empyreus · Pull Request #873 · microsoft/mscclpp

RJ Souza (Empyreus) · 2026-08-13T04:09:39Z

Add pipeline that runs on gb200 nodes, this allows for testing of mnnvl functionality and performance.

… into rjsouza/gb200-ci

azure-pipelines · 2026-08-13T04:09:48Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Pull request overview

Adds a GB200 MNNVL Azure pipeline for deployment, unit testing, and collective benchmarking.

Changes:

Adds GB200 deployment and test templates.
Adds GB200 benchmark tuning support.
Introduces proxy SSH, IMEX, and multi-architecture image handling.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`test/deploy/setup.sh`	Formatting-only updates.
`test/deploy/run-remote.sh`	Adds proxy SSH and optional library-path setup.
`test/deploy/deploy.sh`	Adds pilot, proxy, IMEX, and AppArmor handling.
`python/mscclpp_benchmark/tuning_config.py`	Recognizes GB200 GPUs.
`python/mscclpp_benchmark/bench_collective.py`	Enables NVLS candidates for GB200.
`docker/build.sh`	Revises multi-architecture manifest instructions.
`.azure-pipelines/templates/ut-gb200.yml`	Adds GB200 unit-test workflows.
`.azure-pipelines/templates/deploy.yml`	Adds pilot deployment support.
`.azure-pipelines/multi-nodes-test.yml`	Formatting-only update.
`.azure-pipelines/mnnvl-test.yml`	Adds the MNNVL pipeline.

Suppressed comments (2)

test/deploy/deploy.sh:107

This now adds the GB200 IMEX device to every CUDA deployment, including the existing H100/A100 jobs that share this script. Docker rejects --device when the host path does not exist, so those pipelines will fail at container creation. Gate this option to the GB200 deployment (and ensure that flag is passed correctly) while retaining plain --gpus=all elsewhere.

  LAUNCH_OPTION="--gpus=all --device /dev/nvidia-caps-imex-channels/channel0"

test/deploy/deploy.sh:120

The AppArmor workaround described above is GB200/kernel-specific, but this applies apparmor=unconfined to every default CUDA and ROCm test container. That unnecessarily removes AppArmor isolation from unrelated CI jobs. Scope the option to the affected GB200 deployment instead of changing the security profile globally.

  SECURITY_OPTION="--security-opt apparmor=unconfined"

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

    targetType: filePath
    filePath: test/deploy/deploy.sh
-    arguments: ${{ parameters.deployArgs }} ${{ parameters.containerName }} ${{ parameters.sglangImage }}
+    arguments: ${{ parameters.deployArgs }} ${{ parameters.containerName }} ${{ parameters.sglangImage }} ${{ parameters.pilot }}


+SSH_EXTRA_ARGS="-i ${KeyFilePath}"
+if [ -n "${SSH_PROXY_JUMP:-}" ]; then
+  PROXY_KEY_OPT=""
+  [ -n "${SSH_PROXY_KEY:-}" ] && PROXY_KEY_OPT="-i ${SSH_PROXY_KEY} "
+  SSH_EXTRA_ARGS="${SSH_EXTRA_ARGS} -o ProxyCommand=\"ssh ${PROXY_KEY_OPT}-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -W %h:%p ${SSH_PROXY_JUMP}\""


+- name: hostfile
+  type: string
+  default: '$(System.DefaultWorkingDirectory)/test/deploy/hostfile'
+# Container-side hostfile passed to mpirun (one ip per line).


Binyang Li (Binyang2014) · 2026-08-14T16:08:28Z

Revert this change?

Binyang Li (Binyang2014) · 2026-08-14T16:08:50Z

Revert this one?

Binyang Li (Binyang2014) · 2026-08-14T16:09:32Z

 if $USE_DOCKER; then
    INNER="set -euxo pipefail;"
    INNER+=" cd /root/mscclpp;"
+    INNER+=" [ -f /root/mscclpp/.ldpath ] && source /root/mscclpp/.ldpath;"


Why need this?

Binyang Li (Binyang2014) · 2026-08-14T16:20:50Z

+  - task: Bash@3
+    displayName: Generate deploy files
+    inputs:
+      targetType: 'inline'
+      script: |
+        set -e
+        DEPLOY_DIR="$(System.DefaultWorkingDirectory)/test/deploy"
+        # First NVLink pair (2 nodes) from hostEntries; use $1 (the IP) so the
+        # names resolve inside the container too (the DSM hostnames are only in
+        # the agent's /etc/hosts). Bump to more nodes by parsing further lines.
+        NODE0=$(awk 'NF>=2{print $1; exit}'           <<< "${{ parameters.hostEntries }}")
+        NODE1=$(awk 'NF>=2{c++} c==2{print $1; exit}' <<< "${{ parameters.hostEntries }}")
+
+        echo "Host ${NODE0}
+          Port 22345
+          IdentityFile /root/mscclpp/sshkey
+          StrictHostKeyChecking no
+        Host ${NODE1}
+          Port 22345
+          IdentityFile /root/mscclpp/sshkey
+          StrictHostKeyChecking no" > "${DEPLOY_DIR}/config"
+
+        printf '%s\n%s\n' "azhpcuser@${NODE0}" "azhpcuser@${NODE1}" > "${DEPLOY_DIR}/hostfile"
+
+        printf '%s\n%s\n' "${NODE0}" "${NODE1}" > "${DEPLOY_DIR}/hostfile_mpi"
+
+        # Publish the head node so the run steps can target it without hardcoding.
+        echo "##vso[task.setvariable variable=headNode]${NODE0}"


We use only 2 nodes here? But provide 4 nodes?

Binyang Li (Binyang2014) · 2026-08-14T16:21:45Z

+      displayName: Run Collective Benchmarks
+      runRemoteArgs: '--hostfile $(System.DefaultWorkingDirectory)/test/deploy/hostfile --host $(headNode) --user azhpcuser'
+      remoteScript: |
+        mpirun --allow-run-as-root --bind-to numa -hostfile /root/mscclpp/test/deploy/hostfile_mpi -mca btl_tcp_if_include 10.0.5.0/24 -np 8 -npernode 4 \


Can we not hard code 10.0.5.0/24?

Binyang Li (Binyang2014) · 2026-08-14T16:25:36Z

+# and MultiProcessTest.* (Prelim / HostName / HostHash).
+- name: crossNodeFilter
+  type: string
+  default: '-Communicator,Executor,Ib,MemoryChannel,PortChannel,SemaphorePerf,SwitchChannel,TimeoutWithId'


Why exclude MemoryChannel,PortChannel,SemaphorePerf,SwitchChannel?

RJ Souza (Empyreus) and others added 30 commits July 17, 2026 09:31

initial work

2083df3

replace multi-node to test

db96d1c

update deploy and run-remote

db20006

test deploy

a21bd41

test

72333e0

add variable group

7cb7661

test

c5b1197

add diag

e121f2a

update agent pool

cd61b32

test

49a1bda

remove jumpbox

2e64ee0

test

8bd5ae8

add jumpbox

bcbeead

test

85333b1

remove jumpbox

810cd19

test

030a848

fix user name

e67085f

test

09df808

remove cuda12

cb06a64

aarch64 container image

2ff6758

aarch64

e6c53eb

test

cb1cd6b

test

113f2ae

revert setup.sh changes

33de002

remove docker image if it exists already

442fa36

add gb200 for bench collective

c7dfe93

setup test

5f67e49

test

283bc6c

fix hostfile

92916a7

test

6e388e5

RJ Souza (Empyreus) and others added 23 commits August 7, 2026 13:45

add python

bc66427

test

fbc6285

use venv

8392cb2

update build.sh notes

c4c188b

imex fix

cab622c

apparmor unconfined

353d323

fix allreduce algo

a47344f

test

7542da0

add mp ut

ca078a7

test

f2a8ec9

Merge branch 'rjsouza/gb200-ci' of https://github.com/microsoft/mscclpp…

649d2cd

… into rjsouza/gb200-ci

test

45fe34f

test

304ae99

test

8ad95fe

test

bdb08c8

working on ut

c006015

add unit tests

d4d9c50

test

7fe103c

add port to ut

0134b99

work on UT

d89b7db

test

9be4c2f

revert multinode

42b6c56

Merge branch 'main' into rjsouza/gb200-ci

ae83d0e

Binyang Li (Binyang2014) requested review from a team and a balanced review from Copilot August 14, 2026 16:06

Copilot started reviewing on behalf of Binyang Li (Binyang2014) August 14, 2026 16:07 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Binyang Li (Binyang2014) reviewed Aug 14, 2026

View reviewed changes

Navigation Menu

Uh oh!

add mnnvl pipeline using gb200 nodes - #873

add mnnvl pipeline using gb200 nodes#873
RJ Souza (Empyreus) wants to merge 53 commits into
mainfrom
rjsouza/gb200-ci

RJ Souza (Empyreus) commented Aug 13, 2026

Uh oh!

azure-pipelines Bot commented Aug 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Binyang Li (Binyang2014) Aug 14, 2026

Uh oh!

Binyang Li (Binyang2014) Aug 14, 2026

Uh oh!

Binyang Li (Binyang2014) Aug 14, 2026

Uh oh!

Binyang Li (Binyang2014) Aug 14, 2026

Uh oh!

Binyang Li (Binyang2014) Aug 14, 2026

Uh oh!

Binyang Li (Binyang2014) Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

RJ Souza (Empyreus) commented Aug 13, 2026

Uh oh!

azure-pipelines Bot commented Aug 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Binyang Li (Binyang2014) Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Binyang Li (Binyang2014) Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Binyang Li (Binyang2014) Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Binyang Li (Binyang2014) Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Binyang Li (Binyang2014) Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Binyang Li (Binyang2014) Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants